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SCOPE NOTE 


During 10-12 May, 2004, | convened the analyst and manager corps of the 
Directorate of Intelligence for a mandatory, half-day “stand-down” session on 
analytic tradecraft. As indicated in my opening remarks, the state of analysis in the 
Directorate is strong but with room for improvement. The goal in asking every officer 
to come “off line” for a half-day was therefore two-fold: to underscore the importance 
of the Directorate’s analytic mission and to provide a collective opportunity to learn 
and improve by reviewing both the requirements for sound analytic practices and 
lessons learned from poor tradecraft. 


In these sessions, jointly organized by the Sherman Kent School and the Office of 
Policy Support, presentations by senior DI officers addressed the trademarks of 
quality analysis and offered exemplars of strong and weak practices. Many thanks 
to those involved for their illuminating and candid presentations. In order to capture 
the lessons identified and make them readily accessible as an ongoing reference 
i rman Kent School 
as distilled the key concepts from these presentations for 


both hardcopy and web-based dissemination. 


No topic could be more appropriate) 
than this booklet, prepared by a Senior Analytic Service analyst. It is not an 
exhaustive rendering of all that was covered during the refresher sessions but 
provides a synthesis that covers the essential elements. It will assist analysts in 
continuous tradecraft learning and in attaining the highest standards of our 

shared profession. 


] = S E 
Deputy Director for Intelligence 
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TO, 


MESSAGE FROM THE DDI ON THE MAY 2004 “STAND-DOWN” 


In holding these sessions, we are following through on a commitment | made in my 
February speech on the State of Analysis in the Directorate to hold a Tradecraft 
Refresher Course for all Di analysts and managers. As | said in February, 
intelligence analysis is our profession and it is our craft. As practitioners of that 
craft, it is up to us—not others—to ensure that we are continuously learning and 
improving and that we do not lose sight of the fundamentals of our work. 


Our generation of analysts and managers has an especially heavy burden of 
responsibility. First, we face an exponentially growing volume of information 

that must be read, digested, and analyzed. Second, the time it takes to send a 
requirement to the field, mount collection, disseminate and write reports, and deliver 
finished intelligence analysis has been radically compressed. And third and most 
importantly, we now have unprecedented access to the President and his most 
senior national security advisers. This means we must meet a higher standard than 
ever before—a standard set first and foremost by our own professional commitment 
to excellence, but also one expected and demanded by the Administration, the 
Congress, and the American public. 


| called for a “stand-down” because, quite simply, it is overdue and it is needed. In 
my February speech, | characterized the Directorate’s “state of analysis” as strong, 
with room for improvement. And that is why these sessions are being held—to 
improve. We will not get better without trying to get better—and without focusing on 
our mistakes and learning from them. 


Our mission and our corporate responsibility must be to give the policymaker 

the best-crafted analytical product possible. And, when that product leaves CIA, 

it represents and reflects on every one of us. One key message of these “stand- 
down” sessions is that no analyst and no analytic manager can be either a 
passive recipient of intelligence or a disengaged producer of intelligence. We have 
to cultivate a passion for this work, and every one of us must accept personal 
accountability for the quality of his or her work. Moreover, this is not something to 
be feared; it is what gives us our greatest professional satisfaction—taking pride 
in our work. 


In these sessions, analysts and managers will hear some of the challenges we face 


2 = (b)(2 


) 
We have launched a number of initiatives to ensure that those (b) 


Obstacles are overcome. But our primary focus now is the importance of continuous 
improvement. Our goal is not to produce analysis that our customers agree with or 
necessarily even like. Our job is to be objective, cogent, and provide value-added 

to the policy debate over important national security issues. When the Directorate 

is delivering a difficult or unwelcome message, we need to be sure there is a solid 
foundation of credibility with our consumers so they will hear the message. Products 
based on strong tradecraft will stand up well to the scrutiny of skeptical readers. 
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Deputy Director for Intelligence (b)(6) 
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TO, _ 


CONTINUOUS LEARNING IN 
THE DI: MAY 2004 REVIEW 
OF ANALYTIC TRADECRAFT 
FUNDAMENTALS 


or such impact on the highest levels of 
our government. As the DCI said earlier 
this year, we will never be “all right or 
all wrong.” When problems occur, it is 
important for Di managers and analysts 


| to reflect on what we have done right 

| and what we can do better. A period 

| of self-examination is important now 
and can serve as one of the best ways 
of constantly working to improve the 
“quality of our analysis.” Like any catchall 

| phrase, this concept is shorthand for 

| a lot of different things. In its broadest 

| sense, it requires: 


“This is a difficult profession. You are 
asked to inform the debate on some 

of the country’s most important policy 
judgments, usually based on limited 
and conflicting information. We only get 
pieces of the puzzle. It is like trying to 
do a 1,000-piece jigsaw puzzle with only 
200 pieces. And, as a kicker, you do not 
get to see the lid of the box to tell you 
what it is supposed to look like.” 


--DDI, February 2004 


“When we make mistakes we need 

to learn from them collectively as 

a Directorate. We will enhance our 
expertise and broaden our point of view 
by reaching out to others, employing 
contrarian analyses and, perhaps most 
importantly, by expanding the diversity of 
our workforce.” 


--DDI, May 2004 


A Time for Reflection on the 
Quality of Analysis 


The foregoing observations highlight 
the significant responsibilities and 
challenges that the Directorate of 
Intelligence faces in the period ahead 
for providing insightful analysis to our 
nation’s most senior leaders. Seldom 
has the Directorate had such access to 


e Sophisticated, value-added 
analysis on key national security 
issues; it is not “gisting” cables 
but rather putting an issue into a 
bigger context, providing nuance 
and details that will drive a 
critical foreign policy decision 
or discussion. 


e World-class expertise on an 
issue; it is not just opinion but 
insight, informed by a career- 
long development of expertise. 


e Rigorous re-examination of 
analytical judgments and 
assumptions; it is not holding 
onto an analytical model or line 
of reasoning just because it has 
proven to be reliable in the past. 


Translating these broad objectives 
into every day practice means 
that DI analysis must contain the 
following attributes: 


e Precision of language. 


e Clear articulation of judgments 
and levels of confidence. 


fi okt is 
important for 

DI managers 
and analysts to 
reflect on what 
we have done 
right and what 
we can do better. 
A period of self- 
examination is 
important...” 
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“Continuous 
learning will 
require new 
tradecraft review 
processes...” 


“The current 
focus on Iraq 
should not 
mislead analysts 
. . . analytical 
controversy 
could well 

visit others in 
the future.” 


Understanding and explaining 
the quality of intelligence 
sources and key gaps. 


e Examination of alternative 
analytical possibilities 
or outcomes. 


Key Principles to Guide 
DI Analysis 


Maintaining high-quality analysis 
demands that the DI understand and 
put into practice daily some important 
principles that should shape everything 
we do. Among these are: continuous 
learning, personal accountability, 
corporate responsibility, analytical 
integrity, precision and accuracy, and 
insightful analysis. 


Continuous Learning. The DDI has 
noted that any organization that does 
not learn from its mistakes is destined 
to repeat them. While the DI has had 
numerous successes, it has also made 
mistakes and will make mistakes again. 
When we do, we will admit them, study 
them and learn from them. As a senior 
manager remarked, “the East India 
Company—a global power in the 18" 
century, exists today but only in the 
history books. If we do not constantly 
reassess our tradecraft, we too could 
go the way of the East India Company.” 
As practitioners of that craft it is up to 
us—not others—to ensure that we are 
continuously learning and improving. We 
must not lose sight of the fundamentals 
of our work—the key elements of our 
tradecraft. We will not get better without 
trying to get better. 


Continuous learning will require new 
tradecraft review processes to develop 
understanding of where our analysis 
can improve and where our analytical 
workforce can be assisted. The Iraq 
WMD Review Group, established 


in 2003, was one mechanism for 
examining critical judgments and 
incorporating lessons learned into 
the Directorate’s training programs. 


will allow us to examine 
_ Our tradecraft more systematically, 


and turn those findings into 

~ new tradecratt training objectives that 
will strengthen the Directorate. Already, 
the DI has initiated an “Advanced 
Analyst Training Program” that will be 
offered in Fall 2004 and will draw heavily 
from our lessons learned to bolster 

best practices. 


Personal Accountability. The current 
focus on Iraq should not mislead 
analysts to think they are not personally 
affected by tradecraft mistakes. Analysts 
who worked on the Soviet Union or 
Central America in the 1980s, 
India-Pakistan nuclear issues in the 
mid-1990s, and terrorism since 9/11 
know that analytical controversy could 
well visit others in the future. Rigorous 
testing of assumptions and revisiting 

of past judgments can prepare each 
analyst for the day when his or her issue 
becomes the focus of policymakers’ 
scrutiny and challenge. An analyst 
prepared for such controversy is an 
analyst who has identified key analytical 
assumptions, has a clear understanding 
of the strengths and weaknesses of 

the reporting, and can articulate how _ 
judgments were reached. | 


dose of 
analytical humility—that is, graciously 
accepting the comments and criticisms 
from colleagues—is a professional 
requirement. Analytical arrogance in 
the coordination process has no place 
in the DI and is dangerous as it leads 
to dismissing valuable information and 
insights into our work, | 
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“We have the 


| Corporate Responsibility. Our obligation to (b)(3 


Senior mentoring is now 
a key skill to insure the DI’s continuous 
learning and ultimate survival as a 
world-class analytical service. 


The pressure to “get it right’ should 
not lead analysts to shy away from 
reaching firm judgments. Mistakes 
will happen; they should teach analysts 
how to improve, not how to avoid 
tough calls. Analysts must take 
responsibility for getting the facts 
straight. Relying on other analysts or 
reviewers to catch errors is a recipe 
for failure and rebuke by senior 
policymakers who notice them. 


judgments are not our own but 
rather are the Agency's. We have the 
obligation to coordinate our products 
fully across the DI and increasingly 
across the Intelligence Community. 
Just as no analyst can assume his 
or her issue is safe from political 
controversy or analytical challenge, 
the entire Directorate must share in 
the responsibility for each and every 
analyst's judgments. That means an 
individual analyst or issue team’s work 
cannot be done in a vacuum. There 
must be dialogue among analysts 
and across issue teams to maximize 
the talents of our diverse and 


coordinate our 
products fully 
across the DI...” 


— sophisticated experts. There must be 


involvement of senior managers as well (b 
as the Senior Analytical Service in this ( 
analytical dialogue, to take responsibility 

for nurturing newer analysts and 

developing their skills as self-critics 

and skeptics. In sum, colleagues must 

seriously read and comment on each 

other's drafts; managers must rigorously 

develop and challenge analysts; senior 

Agency managers from the DCI, DDCI, 

and DDI on down must be prepared 


| to stand behind our judgments to the 


President, the Congress, and the 


| American people. 


Analytical Integrity. The Directorate 


| exists for one reason—to analyze all- 


source information in as even-handed 


| a fashion as is humanly possible and to 


provide policymakers with judgments 
not effected by any policy agenda. The 


| best minds and information in the world 


| will be useless if DI managers and 


analysts are not able to present frank 
assessments to powerful leaders who 
may not agree with our judgments. It 
sounds easy but it is not. Policymakers, 
who rightfully have a policy agenda, 
can make it hard for DI analysts to bring 


J them bad news. They can demand that 


we re-examine the evidence, delineate 
our analytical approach, or look for 
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“ .. we must 
be prepared 
to explain our 
reasoning 
and challenge 
ourselves to 
ensure we are 
not missing 
something.” 


“Credibility 
with senior 
policymakers 
begins with 
getting the 
facts right.” 


“... the 
Directorate has 


to “ask the right 
question’ if it is 
to produce useful 


and insightful 
analysis.” 


_challenge remains the same. 


alternative scenarios. As a Directorate, 
we must be prepared to explain our 
reasoning and challenge ourselves to 
ensure we are not missing something. 
However, at the end of the day, we must 
be prepared to stick to our judgments 
when they are sound or admit to errors 
and correct them as appropriate. 


Analytical integrity is not a new 
challenge but rather an enduring one. 
Richard Helms’s memoirs recount that 
the Special Assistant for Vietnamese 
Affairs in the late 1960s had worked 
himself into exhaustion while briefing 
unpleasant assessments to President 
Lyndon Johnson. Vice President 
Humphrey later praised this senior 
officer for having “served the President 
well by holding your ground and telling 
us about the situation as you saw it 

in Vietnam. It was brutally frank and 
forthright analysis.” And today, the 


also to be open to the 
idea that we could be wrong and will 
admit our mistakes.” Doing so makes 
policymakers respect the analyst and 
the institution. It is what demonstrates 
the Directorate’s integrity and validates 
its mission. 


Precision and Accuracy. Credibility 
with senior policymakers begins with 
getting the facts right. There is a need 
for absolute accuracy and precision 

in everything the Directorate writes 
and briefs. Failing to do that not only 
leads to incorrect judgments, but also 
seriously undercuts our credibility with 


A our readers. Misquoting 


will be noticed and, 


if not corrected, foster a belief among 
policymakers that they cannot depend 
on CIA fact-checking. 


agenda, and the state of == 
i interagency process. 


Choosing our words carefully so that 
the reader grasps the exact meaning 
intended is equally important. Analysts 
should not confuse elegant prose 

for precision and clarity. Caveats or 
qualifications lost in the editorial review 
process can distort the analyst's 
intended message and mislead the 
reader. Explaining that some nuance 
had been removed by a night editor is 
not likely to mollify an annoyed reader 
the day after. Analysts must be prepared 
to go the extra mile in articulating the 
importance of caveats and context to 
editors who do not know the issues as 
well as the experts, and they must be 
willing to stand by their analysis. 


Insightful Analysis. The mission of 
preparing insightful analysis is what 
distinguishes the Directorate from any 
other organization to which senior US 
policymakers could go for information 
and analysis. The New York Times, 

The Economist, or Foreign Affairs can 
write about whatever they decide will be 
controversial, appealing, or profitable. 
The CIA, however, is in the business of 
telling a select, demanding, and decisive 
group of policymakers what it does not 
already know, will find useful, and will 
find credible. Simply put, the Directorate 
has to “ask the right question” if it is to 
produce useful and insightful analysis. 
Providing obvious conclusions will cause 
our work to be discounted and make our 
jobs harder the next day. In order to ask 
the right intelligence question, analysts 
need to know the audience, the ‘policy 


e Analysis that is irrelevant 
or ignored comes from 
targeting the wrong audience 
or misunderstanding what 
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the audience's level of 
understanding and policy 
needs are. 


e insightful or “value-added” 
analysis comes from knowing 
what the policymaker already 
knows, what he cannot yet 
know, and what he would find 
useful in understanding a | 
complex story. | 

| 


Ultimately, the target audience—the 


“Persuading the 
policymaker that 
you are right is 
your job...” 


“Understanding 
where the 
policymaker is 
in the decision- 
| making process 
will help...” 


President, an Assistant Secretary, or a 
military commander—will be the judge A Policymaker’s Top Ten 
of whether CIA analysis has been useful for Analysts 

or credible. CIA team chiefs, publication 


editors, or IC colleagues will review --Get me information and analysis 
this analysis, but they are NOT the on time, so | can make an informed 
audience. Persuading the policymaker decision. If it is late, | will decide 
that you are right is your job, not the without it. 
policymaker’s responsibility. This 
challenge warrants the additional --Tell me something I do not already 
elaboration that follows. know. Know what | know and what 

| need. 
Crafting Insightful, 
Persuasive Analysis --Give me fact-based analysis, and let 


others write op-ed pieces. You need 


CIA analysts m whatthe to organize facts in a way that reveals 
_ audience knows patterns, notes change, and has insight. 
~ [Understanding  --What are alll the angles on the issue? 


where the policymaker is in the decision- What did | not ask for that | would need 
making process will help determine the to understand the issue? 
kind of analysis that is needed: 


7 --Have you convinced me that you 

|have examined all the alternatives? You 
should know all the reasons for doubting 
your conclusions. 

|--Is your argumentation transparent? 

| You must tell me what you know, where 
|your data come from, and whether you 
are confident or not. Tell me if you are 
‘changing your line of argument and why. 
| 
|--What should worry me, but also what 
opportunities are there? If you over- 
warn, | will ignore you to our peril. 


Approved for Release: 2021/09/21 C01179397 


(b)(2) 
(b)(3) 


ee 


“... Caveats 
are essential 
and help 
policymakers 


gauge how much 
trust to put in our 


analysis.” 


“,.. Our analysis 
becomes more 
insightful when 
it goes beyond 


answering the 
immediate 
question and 


examines related 


issues...” 
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--Keep me honest with any bad news | 
but you will have to persuade me. If | 
misunderstand an issue, figure out a way 
to make me listen. 


--Check for your own policy bias and 
be open to your own fallibility. Do not let 
your personal views color your analysis 
or blind you to contrary information. 


--lf you are wrong tell me so. If you do, | 


am more likely to respect you and have 
confidence in you than if you do not. 


Useful information can be conveyed 


__in many forms. It may be a fact 


br 


| 


_rather an unclassified description 


a araphic that displays a complex 
rocess ona 
single page. Sometimes the most 


useful information is not a secret, 


Once the analyst understands the 
audience's needs, the challenge is 

to present facts and argumentation 

in a way that is clear, accurate, and 
compelling. Analysis that has no clear 
focus, is missing details, and has not 
carefully weighed the evidence or 
assessed information gaps cannot be 
fixed by editors. The goal of sending 
forward well-supported argumentation 
does not mean, however, overstating a 
case or ignoring problems of sourcing or 
contradictory reporting. 


Caveats are essential and help 
policymakers gauge how much trust to 
put in our analysis and how much risk 
they run in basing policies on it. It is 
the analyst's responsibility to know the 


certainty of the information | 


Finally, our analysis becomes more 
insightful when it goes beyond 
answering the immediate question and 
examines related issues, which the 
policymaker must also be aware of in 
order to appreciate the key conclusions 
of our analysis. That is what will 
distinguish insightful analysis from the 
obvious. Answering the policymaker’s 
first question may also suggest other 
areas where the analyst should provide 


further ate ee = 
the issu 


(b)(2) 
(b)(3) 


even ifitisonlyamental = 


‘exercise that Captures the broader 
context in which the immediate issue sits 
and connects it to other issues important 
to the target audience. 
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must earn its 
| reputation 
| every day...” 


“We are paid to 
make the tough 
calls.” 


[i — e — 


| These simple, yet important, steps to 

| | improved tradecraft are a beginning 

for a revitalized DI commitment to 

| personal and corporate responsibility 

for producing high quality, insightful 

analysis. As the Directorate enters 

| its second half-century in supporting 

| American foreign and security policies, 

we must rededicate ourselves to the 

notion that the Directorate must earn 

its reputation every day as the world’s 

| best analytical service, and that 

| requires continuous learning from 
past efforts. 


| Final Thoughts from the DDI 


| The demands and expectations on us 
are enormous, but we must recognize 
that they are a result of our success. 
What we do matters. There are some 
who say we should not try to make the 
tough calls. However, we cannot avoid 
making a judgment because we are 

| afraid we will be wrong. We are paid 

| to make the tough calls. That is what a 
professional intelligence analyst does. 
Most often, this means relying upon 

| our expertise, reporting, and tradecraft 
to come down on one side or the other 
of an argument; at times, however, 

this may mean acknowledging that 
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“++ we must 
give the 
policymaker full 
transparency into 
our confidence in 
the judgments.” 


the reporting could support several 
hypotheses. But we must remember that 
when we “call it as we see it,” we must 
give the policymaker full transparency 
into our confidence in the judgments. We 
must be clear, tell them what we know, 
what we do not know, what our judgment 
is, and occasionally—when there is 

no firm reporting—tell them what our 
experience suggests to us. As | said in 
February, “if you apply good tradecraft— 
and do your job to the fullest—you can 
be sure that | will defend your analysis 
before any critic.” 
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